WO 02/075654 A2 11111111111111111111111111111111111110 



(12) INTERNATIONAL APPLICATION PUBLISHED UNDER THE PATENT COOPERATION TREATY (PCT) 



(19) World Intellectual Property Organization 
International Bureau 

(43) International Publication Date 

26 September 2002 (26.09.2002) 




PCT 



in 



IIIDIIIIIIIIIIIIIIU 




(10) International Publication Number 



WO 02/075654 A2 



(51) International Patent Classification 7 : G06T 5/00 

(21) International Application Number: PCT/US02/08642 

(22) International Filing Date: 20 March 2002 (20.03.2002) 

(25) Filing Language: English 

(26) Publication Language: English 

(30) Priority Data: 

09/813,750 20 March 2001 (20.03.2001) US 

(71) Applicant: HEWLETT-PACKARD COMPANY 

[US/US]; 3000 Hanover Street, Palo Alto, CA 94304-1112 
(US). 

(72) Inventor: NGUYEN, Truong-Thao; 5204 Hilltop Court, 
Fort Lee, NJ 07024 (US). 

(74) Agent: HEMINGER, Susan; Hewlett-Packard Company, 
Legal Department, IP Section, P.O. Box 272400, Ft. 
Collins, CO 80527-2400 (US). 



(81) Designated States (national): AE, AG, AL, AM, AT, AU, 
AZ, BA, BB, BG, BR, BY, BZ, CA, CH, CN, CO, CR, CU, 
CZ, DE, DK, DM, DZ, EC, EE, ES, FI, GB, GD, GE, GH, 
GM, HR, HU, ID, IL, IN, IS, IP, KE, KG, KP, KR, KZ, LC, 
LK, LR, LS, LT, LU, LV, MA, MD, MG, MK, MN, MW, 
MX, MZ, NO, NZ, OM, PH, PL, PT, RO, RU, SD, SE, SG, 
SI, SK, SL, TJ, TM, TN, TR, TT, TZ, UA, UG, UZ, VN, 
YU, ZA, ZM, ZW. 

(84) Designated States (regional): ARIPO patent (GH, GM, 
KE, LS, MW, MZ, SD, SL, SZ, TZ, UG, ZM, ZW), 
Eurasian patent (AM, AZ, BY, KG, KZ, MD, RU, TJ, TM), 
European patent (AT, BE, CH, CY, DE, DK, ES, FI, FR, 
GB, GR, IE, IT, LU, MC, NL, PT, SE, TR), OAPI patent 
(BF, BJ, CF, CG, Cl, CM, GA, GN, GQ, GW, ML, MR, 
NE, SN, TD, TG). 

Published: 

— without international search report and to be republished 
upon receipt of that report 



[Continued on next page] 



(54) Title: SYSTEM AND METHOD FOR ASYMMETRICALLY DEMOSAICING RAW DATA IMAGES USING COLOR DIS- 
CONTINUITY EQUALIZATION 



100 

1 




(57) Abstract: A system and method for demosaicing raw data ("mosaiced") images utilizes an asymmetric interpolation scheme to 
equalize color discontinuities in the resulting demosaiced images using discontinuities of a selected color component of the mosaiced 
images. Discontinuities of the selected color component are assumed to be equal to discontinuities of the other remaining color 
components. Thus, color discontinuity equalization is achieved by equating the discontinuities of the remaining color components 
with the discontinuities of the selected color component. The asymmetric interpolation scheme allows the system and method to 
reduce color aliasing and non-colored "zippering" artifacts along feature edges of the resulting demosaiced images, as well as colored 
artifacts. 





WO 02/075654 A2 



For two-letter codes and other abbreviations , refer to the "Guid- 
ance Notes on Codes and Abbreviations" appearing at the begin- 
ning of each regidar issue of the PCT Gazette . 





WO 02/075654 



PCT/US02/08642 



SYSTEM AND METHOD FOR ASYMMETRICALLY DEMOSAICING RAW 
DATA IMAGES USING COLOR DISCONTINUITY EQUALIZATION 

5 FIELD OF THE INVENTION 

The invention relates generally to the field of image processing, and 
more particularly to a system and method for demosaicing raw data 
(mosaiced) images. 

10 

BACKGROUND OF THE INVENTION 

Color digital cameras are becoming ubiquitous in the consumer 
market place, partly due to progressive price reductions. Color digital 
is cameras typically employ a single optical sensor, either a Charge Coupled 
device (CCD) or a Complementary Metal Oxide Semiconductor (CMOS) 
sensor, to digitally capture a scene of interest. Both CCD and CMOS 
sensors are only sensitive to illumination. Consequently, these sensors 
cannot discriminate between different colors. In order to achieve color 
20 discrimination, a color filtering technique is applied to separate light in 
terms of primary colors, typically red, green and blue. 

A common filtering technique utilizes a color-filter array (CFA), 
which is overlaid on the sensor, to separate colors of impinging light in a 
Bayer pattern. A Bayer pattern is a periodic pattern with a period of two 
25 different color pixels in each dimension (vertical and horizontal). In the 
horizontal direction, a single period includes either a green pixel and a red 
pixel, or a blue pixel and a green pixel. In the vertical direction, a single 
period includes either a green pixel and a blue pixel, or a red pixel and a 
green pixel. Therefore, the number of green pixels is twice the number of 
30 red or blue pixels. The reason for the disparity in the number of green 
pixels is because the human eye is not equally sensitive to these three 
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colors. Consequently, more green pixels are needed to create a color 
image of a scene that will be perceived as a “true color” image. 

Due to the CFA, the image captured by the sensor is therefore a 
mosaiced image, also called “raw data" image, in which each pixel of the 
5 mosaiced image only holds the intensity value for red, green or blue. The 
mosaiced image can then be demosaiced to create a color image by 
estimating the missing color values for each pixel of the mosaiced image. 
The missing color values of a pixel are estimated by using corresponding 
color information from surrounding pixels. 

10 Although there are a number of conventional demosaicing methods 

to convert a mosaiced image into a color (“demosaiced") image, the most 
basic demosaicing method is the bilinear interpolation method. The 
bilinear interpolation method involves averaging the color values of 
neighboring pixels of a given pixel to estimate the missing color values for 
15 that given pixel. As an example, if a given pixel is missing a color value 
for red, the red color values of pixels that are adjacent to the given pixel 
are averaged to estimate the red color value for that given pixel. In this 
fashion, the missing color values for each pixel of a mosaiced image can 
be estimated to convert the mosaiced image into a color image. 

20 A concern with the bilinear interpolation method is that the resulting 

color images are prone to colored artifacts along feature edges of the 
images. A prior art demosaicing technique of interest that addresses the 
appearance of colored artifacts utilizes an adaptive interpolation process 
to estimate one or more missing color values. According to the prior art 
25 demosaicing technique, first and second classifiers are first computed to 
select a preferred interpolation, which includes arithmetic averages and 
approximated scaled Laplacian second-order terms for the predefined 
color values. The first and second classifiers can be either horizontal and 
vertical classifiers, or positive-slope diagonal and negative-slope diagonal 
30 classifiers. The classifiers include different color values of nearby pixels 
along an axis, i.e., the horizontal, vertical, positive-slope diagonal or 
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negative-slope diagonal. The two classifiers are then compared to each 
other to select the preferred interpolation. 

Although the prior art demosaicing technique of adaptive 
interpolation results in demosaiced color images with reduced colored 
5 artifacts along feature edges, there is still a need for a system and method 
for efficiently demosaicing input mosaiced images to reduce other types of 
artifacts, such as color aliasing and non-colored “zippering” artifacts along 
feature edges. 

10 SUMMARY OF THE INVENTION 

A system and method for demosaicing raw data (“mosaiced”) 
images utilizes an asymmetric interpolation scheme to equalize color 
discontinuities in the resulting demosaiced images using discontinuities of 
15 a selected color component of the mosaiced images. Discontinuities of 
the selected color component are assumed to be equal to discontinuities 
of the other remaining color components. Thus, color discontinuity 
equalization is achieved by equating the discontinuities of the remaining 
color components with the discontinuities of the selected color component. 
20 The asymmetric interpolation scheme allows the system and method to 
reduce color aliasing and non-colored “zippering” artifacts along feature 
edges of the resulting demosaiced images, as well as colored artifacts. 

A method of demosaicing a mosaiced image to derive a 
demosaiced image in accordance with the present includes a step of 
25 independently interpolating first color values of the mosaiced image to 
derive first interpolated values of the demosaiced image and a step of 
interpolating second color values of the mosaiced image to derive second 
interpolated values of the demosaiced image. The step of interpolating 
the second color values includes substantially equalizing a discontinuity of 
30 the second interpolated values with a corresponding discontinuity of the 
first interpolated values. 
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In an embodiment, the step of independently interpolating the first 
color values may include adaptively interpolating the first color values 
using an interpolation technique selected from at least a first interpolation 
technique and a second interpolation technique. The selection of the 
5 interpolation technique may include determining variations of the first and 
second color values along at least a first direction and a second direction, 
such as a horizontal direction and a vertical direction. 

In an embodiment, the step of interpolating the second color values 
includes computing color discontinuity equalization values using 
10 interpolated first color values and averaged first color values. The 

interpolated first color values may be equal to the first interpolated values. 
The color discontinuity equalization values may be derived by subtracting 
the averaged first color values from the interpolated first color values. The 
averaged first color values may be derived by sub-sampling the 
15 interpolated first color values with respect to pixel locations of the 
mosaiced image that correspond to the second color values of the 
mosaiced image to derive sub-sampled values and averaging the sub- 
sampled values to generate the averaged first color values. In this 
embodiment, the step of interpolating the second color values may include 
20 avera ging the second color values of the mosaiced image to derive 

averaged second color values and summing the color discontinuity values 
and the averaged second color values to derive the second interpolated 
values. 

The method may further include a step of selectively compensating 
25 for intensity mismatch between a first type of the first color values and a 
second type of the first color values. In an embodiment, the selective 
compensation includes smoothing the first color values of the mosaiced 
image when gradient and curvature of the first color values are below a 
threshold. The method may also include a step of sharpening the 
30 demosaiced image by operating only on the first interpolated values. 
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A system for demosaicing a mosaiced image to derive a 
demosaiced image includes a first interpolator for independently 
interpolating first color values of the mosaiced image to derive first 
interpolated values of the demosaiced image, a second interpolator for 
5 interpolating second color values of the mosaiced image to derive second 
interpolated values of the demosaiced image, and a color discontinuity 
equalization unit for substantially equalizing a discontinuity of the second 
interpolated values with a corresponding discontinuity of the first 
interpolated values. 

10 In an embodiment, the first interpolator includes an adaptive 

interpolator that is configured to adaptively interpolate the first color values 
using an interpolation technique selected from at least a first interpolation 
technique and a second interpolation technique. In one embodiment, the 
adaptive interpolator includes a gradient direction detector that is 
15 configured to determine variations of the first and second color values 
along at least a first direction and a second direction, such as a horizontal 
direction and a vertical direction. 

In an embodiment, the color discontinuity equalization unit is 
configured to compute color discontinuity equalization values using 
20 interpolated first color values and averaged first color values. The 

interpolated first color values may be equal to the first interpolated values. 
The color discontinuity equalization values may be derived by subtracting 
the averaged first color values from the interpolated first color values. The 
color discontinuity equalization unit may include a sub-sampling unit and 
25 averaging unit. The sub-sampling unit is configured to sub-sample the 
interpolated first color values with respect to pixel locations of the 
mosaiced image that correspond to the second color values of the 
mosaiced image to derive sub-sampled values. The averaging unit is 
configured to average the sub-sampled values to generate the averaged 
30 first color values. In this embodiment, the second interpolator may include 
an averaging unit that is configured to average the second color values of 
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x V. 

the mosaiced image to derive averaged second color values and a 
summing unit that is configured to sum the color discontinuity equalization 
values from the averaged second color values to derive the second 
interpolated values. 

5 The system may further include an intensity mismatch compensator 

that is configured to selectively compensate for intensity mismatch v ' 

between a first type of the first color values and a second type of the first 
color values. In an embodiment, the intensity mismatch compensator may 
be configured to smooth the first color values of the mosaiced image when 
10 gradient and curvature of the first color values are below a threshold. The 
system may also include an image sharpener that is configured to sharpen 
the demosaiced image by operating only on the first interpolated values. 

V ■ , 

Other aspects and advantages of the present invention will become 
apparent from the following detailed description, taken in conjunction with 
is the accompanying drawings, illustrated by way of example of the 
principles of the invention. 

BRIEF DESCRIPTION OF THE DRAWINGS 

V 

20 Fig. 1 is a block diagram of an image processing system in 

accordance with the present invention. 

Fig. 2A illustrates the Bayer pattern of captured intensity values in a 
mosaiced image. 

Fig. 2B illustrates the different color planes of a Bayer-patterned 
25 mosaiced image. 

Fig. 3 is a block diagram a demosaicing unit of the system of Fig. 1 . 

Fig. 4 is a block diagram of a G1-G2 mismatch compensator of the 
demosaicing unit of Fig. 3. 

Fig. 5 is a block diagram of an adaptive interpolator of the 
30 demosaicing unit. 
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Fig. 6 is a process flow diagram illustrating the operation of a G- 
path module of the demosaicing. 

Fig. 7 is a process flow diagram illustrating the operation of a G 
processing block of a color-path module of the demosaicing unit. 

5 Fig. 8 is a process flow diagram illustrating the operation of an R 

processing block of the color-path module of the demosaicing unit. 

Fig. 9 is a block diagram of a G-path module with adaptive 
interpolation and image sharpening capabilities in accordance with an 
alternative embodiment. 

10 Fig. 10 is a block diagram of a G-path module with G1-G2 mismatch 

compensation and adaptive interpolation capabilities in accordance with 
an alternative embodiment of the invention. 

Fig. 1 1 is a block diagram of a G-path module with G1-G2 mismatch 
compensation, adaptive interpolation, and image sharpening capabilities 
15 in accordance with an alternative embodiment. 

Fig. 12 is a block diagram of a G processing block of a color-path 
module in accordance with an alternative embodiment of the invention. 

Fig. 13 is a block diagram of a G processing block of a color-path 
module in accordance with a simplified alternative embodiment of the 
20 invention. 

DETAILED DESCRIPTION 

With reference to Fig. 1, an image processing system 100 in 
25 accordance with the present invention is shown. The image processing 
system operates to digitally capture a scene of interest as a mosaiced or 
raw data image. The mosaiced image is then demosaiced and 
subsequently compressed for storage by the system. The image 
processing system utilizes a demosaicing process based on bilinear 
30 interpolation that reduces color aliasing and non-colored “zippering” 
artifacts along feature edges, as well as colored artifacts. 
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The image processing system 100 includes an image capturing unit 
102, a demosaicing unit 104, a compression unit 106, and a storage unit 
108. The image capturing unit includes a sensor and a color-filter array 
(CFA). The sensor may be a Charged Coupled Device (CCD), a 
5 Complementary Metal Oxide Semiconductor (CMOS) senor, or other type 
of photosensitive sensor. In an exemplary embodiment, the CFA includes 
red (R), green (G) and blue ( B ) filters arranged in a Bayer filter pattern. 
However, the CFA may include filters of other colors arranged in a 
different filter pattern. The CFA operates to allow only light of a particular 
io color to be transmitted to each photosensitive element of the sensor. ^ 

Thus, a digital image captured by the image capturing unit is a mosaiced 
image composed of single-colored pixels that are arranged in a color 
pattern in accordance with the filter patter of the CFA. Consequently, 
each pixel of the mosaiced image has an intensity value for only a single 
15 color, e.g., R, G or B. 

In the exemplary embodiment, the single-colored pixels of the 
mosaiced images acquired by the image capturing unit 102 are arranged i 
in a Bayer pattern due to the configuration of the CFA of the image 
capturing unit. A portion of a mosaiced image in a Bayer pattern is 
20 illustrated in Fig. 2A. Since each pixel of the mosaiced image has an 

intensity value for only a single color, each pixel is missing intensity values 
for the other two colors that are needed to produce a color or demosaiced 
image. As shown in Fig. 2A, the G-colored pixels of the mosaiced image 
are identified as either G1 or G2. Therefore, the mosaiced image of Fig. * 
25 2A can be decomposed with respect to four color components, R, Gl, G2 
and B, as illustrated in Fig. 2B. These decompositions of a mosaiced 
image will sometimes be referred herein as Gl plane 202, G2 plane 204, R 
plane 206 and B plane 208. The Gl and G2 planes are collectively 
referred herein as the G plane. In some sensors, the photosensitive 
30 elements that capture G intensity values at Gl locations can have a 
different response than the photosensitive elements that capture G 
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intensity values at G2 locations. Therefore, the intensity values at G1 and 
G2 locations may have artificial variations due to response differences of 
the photosensitive elements. These artificial variations will be referred 
herein as “ G1-G2 mismatch". 

5 The demosaicing unit 104 of the image processing system 100 

operates to demosaic an input mosaiced image such that each pixel of the 
resulting demosaiced image has intensity values for all three primary 
colors, e.g., R, G and B, to produce a color or demosaiced image. The 
demosaicing unit estimates the missing intensity values for each pixel of 
10 the input mosaiced image by using available intensity values from 
surrounding pixels. The demosaicing unit may also perform image 
sharpening and G1-G2 mismatch compensation. The operation of the 
demosaicing unit is described in detail below. 

The compression unit 104 of the image processing system 100 
is operates to compress a demosaiced image, produced by the demosaicing 
unit 1 04, to a compressed image file. As an example, the compression 
unit may compress a demosaiced image using a DCT-based compression 
scheme, such as the JPEG compression scheme. Although the 
compression unit and the demosaicing unit are illustrated in Fig. 1 as 
20 separate components of the image processing system, these components 
may be integrated in an application specific integrated chip (ASIC). 
Alternatively, the compression unit and the demosaicing unit may be 
embodied as a software program that performs the functions of these units 
when executed by a processor (not shown). 

25 The storage unit 108 of the image processing system 100 provides 

a medium to store compressed image files from the compression unit 106. 
The storage unit may be a conventional storage memory, such as DRAM. 
Alternatively, the storage unit may be a drive that interfaces with a 
removable storage medium, such as a standard computer floppy disk. 

30 The image capturing unit 102, the demosaicing unit 104, the 

compression unit 106, and the storage unit 108 of the system 100 may be 
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included in a single device, such as a digital camera. Alternatively, the 
image capturing unit may be included in a separate device. In this 
alternative embodiment, the functions of the demosaicing unit, the 
compression unit and the storage unit may be performed by a computer. 

5 Turning to Fig. 3, a block diagram illustrating the components of the 

demosaicing unit 104 is shown. As illustrated in Fig. 3, the demosaicing 
unit includes a color separator 302, a G-path module 304 and a color-path 
module 306. The color separator receives a window of observation of an 
input mosaiced image, and then separates the intensity values within the 
10 observation window with respect to color. Thus, the intensity values are 
separated in terms of R, G and B. The R and B intensity values are 
transmitted to the color-path module, while the G intensity values are 
transmitted to both the G-path module and the color-path module. 

Although not illustrated in Fig. 3, the R and B intensity value are also 
15 transmitted to the G-path module, as will be described with respect to Fig. 

5. The G-path module interpolates the G intensity values of the 
observation window to generate interpolated G intensity values (“G’ 
values”) for each pixel within the observation window of the input 
mosaiced image, while the color-path module interpolates the R and B 
20 intensity values to generate interpolated R and B intensity values (“R ' and 
B’ values”) for each pixel within the observation window. Consequently, 
each pixel within the observation window will have R, G and B values to 
produce a demosaiced window that corresponds to the initial window of 
observation. When all the windows of observation of the input mosaiced 
25 image are processed, a complete demosaiced image is produced. In the 
exemplary embodiment, the G-path module also compensates for G1-G2 
mismatch and sharpens the resulting demosaiced image by sharpening a 
given observation window of an input mosaiced image with respect to only 
G intensity values. In addition, the color-path module provides color 
30 discontinuity equalization by taking into consideration information provided 
by the G intensity values. Color discontinuity equalization is a process of 
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estimating a spatial discontinuity of a particular color within a mosaiced 
image by analyzing the discontinuity of another color at the same location 
of the image. 

Color discontinuity equalization is provided by the demosaicing unit 
5 1 04 by assuming that local color discontinuities within images are same 

for each color component. That is, changes in local intensity values are 
. the same for R, G and B intensity values. This assumption can be 
expressed as: 

AR - AG = AB . (1) 

io If the local color discontinuity of G intensity values is considered to be 
available throughout an image, the local color discontinuity of R and B 
intensity values can be expressed with respect to the color discontinuity of 
G intensity values. The local color discontinuity of R can then be 
expressed as: 

is AR = AG, (2) 

where AR = R-R 0 and AG = G-G 0 . In the above equation, Ro and Go 
are the local averages of available R and G intensity values, respectively. 
Similarly, the local color discontinuity of B can be expressed as: 

AB = AG, (3) 

20 where AB = B-B 0 . In equation (3), Bo is the local average of available B 
intensity values. The equations (2) and (3) can be used to estimate R and 
B intensity values that equalizes color discontinuity of G. That is, equation 
(2) can be rewritten as: 

R = R 0 + AG , and (4) 

25 equation (3) can be rewritten as: 

B = B 0 + AG . (5) 

Note that equations (4) and (5) imply that every color value that is 
available from the original mosaiced image is kept untouched, since AG 
equals AR and AB. Equation (4) can be further rewritten as: 

30 R = G + C ro , (6) 
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where C R0 = -G 0 . In equation (6), R is viewed as being equal to G plus 

some color offset correction, Cro ■ This color offset correction is precisely 
equal to the difference between the local averages of G and R. Similarly, 
equation (5) can be rewritten as: 

5 B = G + C B0 , (7) 

where C bo =B 0 -G 0 . 

In practice, the use of equations (6) and (7) for a demosaicing 
process produces color “zippering” artifacts along feature edges of 
demosaiced images. The color offset correction terms are obtained by 
10 comparing the local averages of G and R, and the local averages of G and 
B. However, the compared averages are not extracted from the same 
pixel locations. Thus, the color offset correction based on equations (6) 
and (7) results in a comparison discrepancy, especially in the presence of 
high intensity gradient, such as on a feature edge. However, if G values 
15 are available at all pixel locations, the local averages of G can be 

calculated at any pixel locations. For best accuracy in the calculation of 
Cro, the G values should be extracted from R locations. Similarly, the G 
values should be extracted from B locations in the calculation of Cro- 
Thus, equations (6) and (7) can be modified as: 



R — Rq + AG R , 


(8) 


where AG„ =G-G„ 0 , and 




B = B 0 + AG B , 


( 9 ) 


where A G B = G - G B0 . 





In the above equations, Cro and Cro denote G averages calculated out of 
25 R locations and B locations, respectively. The terms, A Gr and A Gr of 
equation (8) and (9), represent color discontinuity equalization values to 
equalize color discontinuities of the R and B values with the G values. 

The G-path module 304 of the demosaicing unit 104 operates on 
the G intensity values to independently interpolate the G intensity values 
30 and to make the G intensity values available at all Dixel locations. The 
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color-path module 306 of the demosaicing unit utilizes equations (8) and 
(9) to generate interpolated R and B intensity values to produce a 
demosaiced image that has been color discontinuity equalized. 

As shown in Fig. 3, the G-path module 304 of the demosaicing unit 
5 104 includes a GJ-G2 mismatch compensator 308, an adaptive 

interpolator 310 and an image sharpener 312. The GJ-G2 mismatch 
compensator operates selectively smooth intensity value differences at G1 
and G2 pixels caused by G1-G2 mismatch in regions of an input mosaiced 
image where there are low intensity variations. As shown in Fig. 4, the 
io G1-G2 mismatch compensator includes a pixel-wise gradient and 

curvature magnitude detector 402, a G1-G2 smoothing unit 404 and a 
selector 406. The pixel-wise gradient and curvature magnitude detector 
operates to generate a signal to indicate whether a given window of 
observation of an input mosaiced image is a region of low intensity with 
15 respect to G intensity values. The G1-G2 smoothing unit performs a 

convolution using the following mask to smooth the G intensity values of 
the given observation window. 
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20 

The use of the above mask amounts to replacing every G1 input by the 
midpoint value between the considered input and the average of the four 
G2 neighboring values. The same applies to every G2 input with respect 
to their G1 neighbors. 

25 The signal from the pixel-wise gradient and curvature magnitude 

detector 402 and the G1-G2 smoothed G intensity values from the G1-G2 
smoothing unit 404 are received by the selector 406. The selector also 
receives the original G intensity values of the current window of 
observation of the input mosaiced image. Depending on the signal from 
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the pixel-wise gradient and curvature magnitude detector, the selector 
transmits either the G1-G2 smoothed G intensity values or the original G 
intensity values for further processing. 

As shown in Fig. 4, the pixel-wise gradient and curvature 
5 magnitude detector 402 includes a horizontal gradient filter 408, a 
horizontal curvature filter 410, a vertical gradient filter 412, a vertical 
curvature filter 414, and a variation magnitude analyzer 416. The outputs 
of the filters 408-414 are fed into the variation magnitude analyzer. The 
output of the variation magnitude analyzer is fed into the selector 406. 

10 The horizontal gradient and horizontal curvature filters 408 and 410 utilize 
the following masks to derive a horizontal gradient variation value and a 
horizontal curvature variation value for the G intensity values of the 
observation window of the input mosaiced image. 
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horizontal gradient mask horizontal curvature mask 



The vertical gradient and vertical curvature filters 412 and 414 
utilize the following masks to derive a vertical gradient variation value and 
20 a vertical curvature variation value for the G intensity values of the given 
observation window. 
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vertical curvature mask 



25 The variation magnitude analyzer 416 receives the variation values 

from the horizontal and vertical filters 408-414 and selects the highest 
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variation value, which is identified as the maximum intensity variation 
magnitude of the current observation window. The maximum intensity 
variation magnitude is then compared to a predefined threshold to 
determine whether the G1-G2 mismatch compensation is necessary. If the 
5 maximum intensity variation magnitude exceeds the predefined threshold, 
then a signal is transmitted to the selector 406 so that the G1-G2 
smoothed G intensity values are selected. Otherwise, the variation 
magnitude analyzer transmits a different signal so that the original G 
intensity values of the current observation window are selected by the 
10 selector. 

Turning back to Fig. 3, the adaptive interpolator 310 of the (7-path 
module 304 is situated to receive the G intensity values of the current 
observation window of the input mosaiced image from the G1-G2 
mismatch compensator 308. The adaptive interpolator operates to 
15 selectively apply a horizontal interpolation or a vertical interpolation, 

depending on the intensity variations within the observation window of the 
input mosaiced image to estimate the missing G intensity values for R and 
B pixel locations of the window. The underlying idea of the adaptive 
interpolator is to perform the horizontal interpolation when the image 
20 intensity variation has been detected to be locally vertical, or to perform 
the vertical interpolation when the image variation has been detected to be 
locally horizontal. 

As shown in Fig. 5, the adaptive interpolator 310 includes a 
horizontal interpolation unit 502, a vertical interpolation unit 504, a pixel- 
25 wise gradient direction detector 506 and a selector 508. The horizontal 
interpolation unit and the vertical interpolation unit perform a horizontal 
interpolation and a vertical interpolation, respectively, on the G intensity 
values from the G1-G2 mismatch compensator 308 using the following 
masks. 

30 
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The pixel-wise gradient direction detector 506 of the adaptive 
5 interpolator 310 determines whether the results of the horizontal 

interpolation or the results of the vertical interpolation should be selected 
as the interpolated G intensity values. The pixel-wise gradient direction 
detector includes a horizontal G variation filter 51 0, a horizontal non-G 
variation filter 512, a vertical G variation filter 514 and a vertical non-G 
io variation filter 516. The G variation filters 51 0 and 514 operate on the G 
intensity values of the current observation window of the input mosaiced 
image, while the non-G variation filters 512 and 516 operate on the R and 
B intensity values. The variation filters 510-516 utilize the following mask 
to derive horizontal and vertical variation values with respect to the G 
15 intensity values or the non-G intensity values for the current observation 
window of the input mosaiced image. 



[ 1/2 1 1 / 2 ] 
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The pixel-wise gradient direction detector 506 also includes 
absolute value units 518, 520, 522 and 524, summing units 526 and 528 
and a variation analyzer 530. Each absolute value unit receives the 
variation value from one of the variation filters 510-516 and then takes the 
absolute value to derive a positive variation value, which is then 
transmitted to one of the summing units 526 and 528. The summing unit 
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526 adds the positive variation values from the absolute value units 518 
and 520 to derive a horizontal variation value, while the summing unit 
adds 528 the positive variation values from the absolute value units 522 
and 524 to derive a vertical variation value. The horizontal and vertical 
5 values are then evaluated by the variation analyzer. The variation 
analyzer determines whether the horizontal value is greater than the 
vertical value. If so, the variation analyzer sends a signal to direct the 
selector 508 to transmit the results of the horizontal interpolation. If not, 
the variation analyzer sends a different signal to direct the selector to 
10 transmit the results of the vertical interpolation. 

The horizontal value and the vertical value derived by the pixel-wise 
gradient direction detector 506 include curvature and gradient information. 
At a G location of an input mosaiced image, the coefficients of the 
horizontal mask that are involved in the convolution of G intensity values is 
is [1 0-201], while the coefficients of the horizontal mask that are involved 
in the convolution of non-G intensity values, i.e., R and G intensity values, 
is [0-10 1 0]. Therefore, at a G location, the curvature is given by the G 
intensity values and the gradient is given by the non-G intensity values. In 
contrast, at a non-G location, the curvature is given by the non-G intensity 
20 values and the gradient is given by the G intensity values. However, this 
alternating role of the G intensity values and the non-G intensity values 
does not affect the detection of the dominant image variation direction 
since only the sum of the gradient and the curvature is needed. The same 
reasoning applies to the vertical variation value. 

25 Turing back to Fig. 3, the image sharpener 312 of the G-path 

module 304 is situated to receive the interpolated G intensity values from 
the adaptive interpolator 310. The image sharpener operates to improve 
the global image quality by applying the following sharpening mask to only 
the G intensity values of the current window of observation of the 
30 mosaiced image. 
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The overall operation of the G-path module 304 of the demosaicing 
unit 104 is described with reference to Fig. 6. At step 602, the G intensity 
5 values within a window of observation of an input mosaiced image are 
received by the G-path module. Next, at step 604, a smoothing of G1 and 
G2 intensity differences is performed on the received G intensity values by 
the G1-G2 smoothing unit of the G1-G2 mismatch compensator 308 to 
derive G1-G2 compensated intensity values. At step 606, the maximum 
10 intensity variation magnitude of the observation window is determined by 
the variation magnitude analyzer 416 of the G1-G2 mismatch compensator 
from the variation values generated by the horizontal gradient filter 408, 
the horizontal curvature filter 410, the vertical gradient filter 412 and the 
vertical curvature filter 414 of the G1-G2 mismatch compensator. At step 
15 608, a determination is made whether the maximum intensity variation 

magnitude is greater than a predefined threshold. If so, the current 
window of observation is determined to be a region of high intensity 
variation, where G1-G2 mismatch compensation (smoothing of G1 and G2 
intensity differences) should not be performed. Thus, the process 
20 proceeds to step 61 0, at which the original G intensity values are 

transmitted for further processing. However, if the maximum intensity 
variation magnitude is not greater than the threshold, the current window 
of observation is determined to be a region of low intensity variation, 
where G1-G2 mismatch compensation should be performed. Thus, the 
25 process proceeds to step 612, at which the G1-G2 compensated intensity 
values are transmitted for further processing. 

Next, at step 614, a horizontal interpolation is performed by the 
horizontal interpolation unit 602 of the adaptive interpolator 310. Similarly, 
at step 616, a vertical interpolation is performed by the vertical 
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interpolation unit 504 of the adaptive interpolator. Next, at step 618, a 
horizontal variation value is computed by the horizontal G variation filter 
510, the horizontal non-G variation filter 512, the absolute value units 518 
and 520 and the summing unit 526 of the adaptive interpolator. In 
5 addition, a vertical variation value is computed by the vertical G variation 
filter 514, the vertical non-G variation filter 516, the absolute value units 
522 and 524 and the summing unit 528 of the adaptive interpolator. Steps 
614, 616 and 618 are preferably performed in parallel. At step 620, a 
determination is made whether the horizontal variation value is greater 
10 than the vertical variation value. If so, the results of the horizontal 

interpolation are transmitted for further processing, at step 622. If not, the 
results of the vertical interpolation are transmitted for further processing, at 
step 624. Next, at step 626, image sharpening is performed by the image 
sharpener 312 of the G-path module 304 by applying a sharpening mask 
15 to the interpolated G intensity values. 

In the exemplary embodiment, the G-path module 304 of the 
demosaicing unit 104 includes both the G1-G2 mismatch compensator 308 
and the image sharpener 312. However, one or both of these 
components of the G-path module may be removed from the G-path 
20 module. As an example, if G1-G2 mismatch is not a significant factor for 
mosaiced images produced by the image capturing unit 102 of the system 
100, the G1-G2 mismatch compensator may not be included in the G-path 
module. Thus, the G1-G2 mismatch compensator and the image 
sharpener are optional components of the G-path module. 

25 The output of the G-path module 304 of the demosaicing unit 104 is 

a set of final interpolated G intensity values (“G' values”). As will be 
described below, the outputs of the color-path module 306 are a set of 
final interpolated R intensity values (“it' values”) and a set of final 
interpolated B intensity values (“5’ values”). These intensity values 
30 represent color components of a demosaiced color image. Using 
equations (8) and (9), the R’ and B 'values of the demosaiced image 
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produced by the color-path module include color discontinuity equalization 
components that provide discontinuity equalization of the R and B 
components of the demosaiced image with respect to the G component of 
the image. 

5 As shown in Fig. 3, the color-path module 306 of the demosaicing 

unit 104 includes an R processing block 314, a G processing block 316 
and a B processing block 318. Each of these blocks operates on only one 
of the color planes of the current observation window of the input 
mosaiced image. For example, the R processing block operates only on 
to the R plane of the observation window. However, as described below, 
results from the G processing block are used by the R and B processing 
blocks to introduce color correlation between the different color planes for 
color discontinuity equalization. Thus, the G processing block of the color- 
path module is described first. 

15 The G processing block 316 of the color-path module 306 includes 

an adaptive interpolator 320, an R sub-sampling unit 322, a B sub- 
sampling unit 324 and interpolation and averaging filters 326 and 328. 

The adaptive interpolator 320 is identical to the adaptive interpolator 310 
of the G-path module 304. Thus, the adaptive interpolator 320 of the G- 
20 processing block operates on the G intensity values of an observation 
window of an input mosaiced image to adaptively interpolate the G 
intensity values to derive missing G intensity values for the R locations and 
B locations of the observation window. The R sub-sampling unit operates 
to sub-sample the interpolated G intensity values (" Go values”) from the 
25 adaptive interpolator 320 in terms of R locations of the observation 
window. That is, the Go values are sub-sampled at R locations of the 
observation window of the input mosaiced image to derive R sub-sampled 
G 0 values. Similarly, the B sub-sampling unit sub-samples the Go values 
in terms of B locations of the observation window to derive B sub-sampled 
30 Go values. The interpolation and averaging filter 326 then interpolates 
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and averages the R sub-sampled Go values using the following averaging 
mask. 
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Similarly, the interpolation and averaging filter 328 interpolates and 
averages the B sub-sampled Go values using the same averaging mask. 
From the interpolation and averaging filters 326 and 328, the R sub- 
sampled and interpolated Go values (“ Gro values”) are sent to the R 
10 processing block 314, while the B sub-sampled and interpolated Go values 
CGbo values”) are sent to the B processing block 318. 

The operation of the G processing block 316 of the color-path 
module 306 is described with reference to Fig. 7. At step 702, the G 
intensity values within a window of observation of an input mosaiced 
15 image are received by the adaptive interpolator 320 of the G processing 
block. Next, at step 704, an adaptive interpolation is performed on the G 
intensity values by the adaptive interpolator 320. The results of the 
adaptive interpolation is derived from either horizontal interpolation or 
vertical interpolation, depending on the horizontal and vertical variation 
20 values associated with the G intensity values of the current observation 
window. The process then separates into two parallel paths, as illustrated 
in Fig. 7. The first path includes steps 706, 708 and 710. At step 706, the 
Go values are sub-sampled in terms of R locations of the observation 
window by the R sub-sampling unit 322 of the G processing block. The R 
25 sub-sampled Go values are then interpolated and averaged by the 

interpolation and averaging filter 326 to derive Gro values, at step 708. 
Next, at step 710, the Gro values are transmitted to the R processing 
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* 

block 314. The second path includes steps 712, 714 and 716. At 
step712, the Go values are sub-sampled in terms of B locations by the B 
sub-sampling unit 324. The B sub-sampled Go values are then 
interpolated and averaged by the interpolation and averaging filter 328 to 
5 derive Gro values, at step 714. Next, at step 716, the Gbo values are 

transmitted to the B processing block 31 8. Steps 706-710 are preferably * 
performed in parallel to steps 712-716. 

The R processing block 314 of the color-path module 314 includes 
an interpolation and averaging filter 330, a subtraction unit 332 and a 
10 summing unit 334. The interpolation and averaging filter 330 interpolates 
and averages the R intensity values of the current observation window of 
the input mosaiced image using the same averaging mask as the 

v 

interpolation and averaging filters 326 and 328 of the G processing block 
316. The subtraction unit then receives the averaged R values ( u Ro 
15 values”), as well as the Gro values from the interpolation and averaging 
filter 326 of the G processing block. For each pixel of the observation 
window, the subtraction unit subtracts the Gro value from the 
corresponding Ro value to derive a subtracted value (“Ro-GBO value”). 

The summing unit then receives the Ro-Gbo values from the subtraction * 
20 unit, as well as the G’ values from the G-path module 304. For each pixel 
of the observation window, the summing unit adds the Ro-Gbo value and 
the corresponding G’ value to derive a final interpolated R intensity value 
("R ’ value”). These R ’ values represent the R component of the 
demosaiced image. 

25 The operation of the R processing block 314 of the color-path 

module 306 is described with reference to Fig. 8. At step 802, the R 
intensity values within a window of observation of an input mosaiced 
image are received by the interpolation and averaging filter 330 of the R 
processing block. Next, at step 804, the R intensity values are 
30 interpolated and averaged by the interpolation and averaging filter 330. At 
step 806, the Rq values are subtracted by the Gro values from the 
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interpolation and averaging filter 326 of the G processing block 316 by the 
subtraction unit 332 of the R processing block. Next, at step 808, the Rq- 
Gro values from the subtraction unit are added to the G’ values from the 
G path module 304 by the summing unit 334 to derive the R’ values of the 
5 observation window. The R' values are then outputted from the R 
processing block, at step 81 0. 

Similar to the R processing block 314, the B processing block 318 
of the color-path module 306 includes an interpolation and averaging filter 
336, a subtraction unit 338 and a summing unit 340. The interpolation 
10 and averaging filter 336 interpolates and averages the B intensity values 
within a given window of observation of an input mosaiced image using 
the same averaging mask as the interpolation and averaging filter 330 of 
the R processing block. The subtraction unit 338 then receives the 
averaged B values (Bo values"), as well as the Gbo values from the 
is averaging unit 328 of the G processing block 316. For each pixel of the 
observation window, the subtraction unit 338 subtracts the Gbo value from 
the corresponding B 0 value to derive a subtracted value (“Bo-Gbo value"). 
The summing unit 340 then receives the Ro-Gbo values from the 
subtraction unit, as well as the G’ values from the G-path module 304. For 
20 each pixel of the observation window, the summing unit 340 adds the Ro- 
Gbo value and the corresponding G’ value to derive a final B interpolated 
value (“B’ value”). These B’ values represent the B component of the 
demosaiced image. The operation of the B processing block is similar to 
the operation of the R processing block, and thus, is not be described 
25 herein. 

From the above description of the color-path module 306, it can be 
seen that the G processing block 316 and the subtraction and summing 
units 332, 334, 338 and 340 of the R and B processing blocks 314 and 
318 operate to factor in color discontinuity equalization values, i.e., A Gr 
30 and A Gb of equation (8) and (9), into the Ro and Bo values of an input 

mosaiced image to generate R’ and B ’ values that results in a demosaiced 
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image with equalized color discontinuities. The color discontinuity 
equalization values for the R component of the input mosaiced image are 
added to the Ro values by first subtracting Gro values generated by the G 
processing block from the Rq values and then adding the G’ values 
5 generated from the G-path module 304. Similarly, the color discontinuity 
equalization values for the B component of the input mosaiced image are 
added to the Bq values by first subtracting Gro values generated by the G 
processing block from the Ro values and then adding the G’ values 
generated from the G-path module 304. 
io In an ASIC implementation, as well as other types of 

implementations with a limited memory capacity, a desired feature of the 
demosaicing unit 104 is that all the necessary convolutions are performed 
in parallel during a single stage of the image processing. Multiple stages 
of convolution typically require intermediate line buffers, which increase 
is the cost of the system. Another desired feature is that the demosaicing 
unit operates on small-sized convolution windows. The size of the 
convolution window determines the number of line buffers that are 
needed. Thus, a smaller convolution window is desired for reducing the 
number of line buffers. Still another desired feature is the use of 
20 averaging masks that are larger than 3x3. Using 3x3 averaging masks 
produces unsatisfactory demosaiced images in terms of color aliasing. 
Thus, the size of the masks should be at least 5x5. Below are alternative 
embodiments of the G-path module 304 and the G processing block 316 of 
the color-path module 306 that allow the above-described features to be 
25 realized. 

In Fig. 9, a G-path module 902 with adaptive interpolation and 
image sharpening capabilities is shown. The G-path module 902 is 
functionally equivalent to the G-path module 304 of Fig. 3 that includes 
only the adaptive interpolator 310 and the image sharpener 312. As 
30 shown in Fig. 9, the G-path module 902 includes the horizontal 

interpolation unit 502, the vertical interpolation unit 504, the pixel-wise 
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gradient direction detector 506 and the selector 508. These components 
502-508 of the G-path module 902 are the same components found in the 
adaptive interpolator 310 of Fig. 3, which are shown in Fig. 5. The 
components 502-508 operate to transmit the results of the horizontal 
5 interpolation or the results of the vertical interpolation, depending on the 
determination of the pixel-wise gradient direction detector 508. 

The G-path module 902 of Fig. 9 also includes a horizontal 
differentiating filter 904, a vertical differentiating filter 906, a selector 908 
and a summing unit 910. These components 904-910 of the G-path 
10 module operate to approximate the image sharpening performed by the 
image sharpener 312 of the G-path module 304 of Fig. 3. The operation 
of the adaptive interpolator 310 and the image sharpener 312 of the G- 
path module 304 of Fig. 3 can be seen as applying a horizontal or vertical 
interpolation mask on the given G intensity values and then applying a 
15 sharpening mask. The sharpening operation can be interpreted as adding 
a differential component to the non-interpolated intensity values. 
Therefore, the combined interpolation and sharpening mask can be 
decomposed as follows. 
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Therefore, the components 904-910 of the G-path module 902 operate to 
selectively add the appropriate differential component to the interpolated G 
intensity values. The horizontal and vertical differentiating filters 904 and 
906 independently operate on the G intensity values within the given 
observation window of the input mosaiced image using the horizontal and 
vertical differential masks, respectively. The selector 908 transmits either 
the results of the vertical sharpening or the results of the horizontal 
sharpening, depending on the determination of the pixel-wise gradient 
direction detector 506. In one scenario, the results of the horizontal 
interpolation and the horizontal sharpening are combined by the summing 
unit 910 to generate the G’ values. In another scenario, the results of the 
vertical interpolation and the vertical sharpening are combined by the 
summing unit 910 to generate the G’ values. However, the interpolation 
and sharpening performed by the G-path module 902 of Fig. 9 generally 
does not yield the same G’ values generated by an equivalent G-path 
module of Fig. 3 that includes only the adaptive interpolator 310 and the 
image sharpener 312, which would sequentially perform the adaptive 
interpolation and the image sharpening. For the G-path module 902 of 
Fig. 9, the output values of the sharpening convolution are derived from 
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the original G intensity values, while the output values of the sharpening 
convolution for the equivalent G-path module of Fig. 3 are derived from 
either the horizontal interpolated G intensity values or the vertical 
interpolated G intensity values. However, this difference does not produce 
5 significant artifacts in the demosaiced image. 

In Fig. 10, a G-path module 1002 with G1-G2 mismatch 
compensation and adaptive interpolation capabilities is shown. The G- 
path module 1002 is functionally equivalent to the G-path module 304 of 
Fig. 3 that includes only the G1-G2 mismatch compensator 308 and the 
to adaptive interpolator 310. As shown in Fig. 10, the G-path module 1002 
includes the horizontal interpolation unit 502, the vertical interpolation unit 
504, the pixel-wise gradient direction detector 506 and the selector 508. 
These components 502-508 of the G-path module 1002 are the same 
components found in the adaptive interpolator 310 of Fig. 3, which are 
15 shown in Fig. 5. The components 502-508 operate to transmit the results 
of the horizontal interpolation or the results of the vertical interpolation, 
depending on the determination of the pixel-wise gradient direction 
detector 508. 

The G-path module 1002 of Fig. 10 also includes a smoothing and 
20 horizontal interpolation filter 1 004, a smoothing and vertical interpolation 
filter 1006, a selector 1008, a second stage selector 1010, and the pixel- 
wise gradient and curvature magnitude detector 402. The filters 1004 and 
1006 perform both adaptive interpolation and G1-G2 mismatch 
compensation in a single stage process. The filters 1004 and 1006 
25 operate on the G intensity values using the following masks. 
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horizontal interpolation and vertical interpolation and 

G1-G2 smoothing mask G1-G2 smoothing mask 

5 The selector 1 008 transmits the results of either the horizontal 

interpolation and G1-G2 smoothing or the vertical interpolation and G1-G2 
smoothing to the second stage selector, depending on the determination 
by the pixel-wise gradient direction detector. The second stage selector V-. 
also receives the results of either the horizontal interpolation or the vertical 
10 interpolation from the selector 508. The second stage selector transmits 
the output values from the selector 508 or the output values from the 
selector 1008 for further processing, depending on the determination 
made by the pixel-wise gradient and curvature magnitude detector. 

Similar to the G-path module 902 of Fig. 9, the G-path module 1002 
is of Fig. 1 0 generally does not yield the same G’ values generated by an 
equivalent G-path module of Fig. 3 that includes only the G1-G2 mismatch 
compensator 308 and the adaptive interpolator 310, which would 
sequentially perform the G1-G2 smoothing and the adaptive interpolation. 
However, this difference again does not produce significant artifacts in the 
20 demosaiced image. 

In Fig. 1 1 , a G-path module 1 1 02 with G1-G2 mismatch 

v 

compensation, adaptive interpolation and sharpening capabilities is 
shown. The G-path module 1102 is functionally equivalent to the G-path 
module 304 of Fig. 3. As shown in Fig. 1 1 , the G-path module 1 1 02 
25 includes all the components of the G-path module 902 of Fig. 9 and the G- 
path module 1002 of Fig. 10. The components contained in the dotted 
box 1104 are all the components of the G-path module 1002 of Fig. 10. 
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These components 502-508 and 1004-1010 generate the G intensity 
values that are the result of G1-G2 smoothing and the adaptive 
interpolation. The G-path module 1002 of Fig. 1 1 also includes the 
horizontal differentiating filter 904, the vertical differentiating filter 906, the 
5 selector 908 and the summing unit 910. These components 904-910 
generate the horizontal and vertical sharpening differential component 
values. The G intensity values from the second stage selector 1010 are 
added to either the horizontal differential component values or the vertical 
differential component values from the selector 908 by the summing unit, 

10 depending on the determination of the pixel-wise gradient direction 
detector. In order to ensure that G1-G2 mismatches do not corrupt the 
calculation of the differential component values, the following differential 
masks are used by the horizontal and vertical differentiating filters 904 and 
906. 
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horizontal differential mask vertical differential mask 

These masks are designed to only "read” either the G1 values or the G2 
20 values. In general, G1-G2 mismatches result in an offset between the 
local averages of G1 and G2, respectively. However, G1-G2 mismatches 
contribute little to the discrepancies between their respective variations. 
Consequently, the outputs of such masks are generally insensitive to Gl- 
G2 mismatches. 

25 In Fig. 12, a G processing block 1202 for the color-path module 306 

of Fig. 3 in accordance with an alternative embodiment of the invention 
(for ASIC implementation) is shown. The G processing block 1202 is 
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functionally equivalent to the G processing block 316 of the color-path 
module 306 of Fig. 3. As shown in Fig. 12, the G processing block 1202 
includes a G1-G2 separator 1204 that separates the G intensity values into 
Gl and G2 values. The G processing block 1202 further includes a 
5 horizontal interpolation and averaging filter 1206, a vertical interpolation 
and averaging filter 1210, the pixel-wise gradient direction detector 506, 
and a selector 1214. These components operate to produce Gro values 
for a given observation window of an input mosaiced image. The G 
processing block also includes a horizontal interpolation and averaging 
10 filter 1212, a vertical interpolation and averaging filter 1208, and a selector 
1216. These components along with the pixel-wise gradient direction 
detector 506 operate to produce Gro values for the current observation 
window. 

The horizontal interpolation and averaging filter 1206 utilizes the 
15 following mask on the Gl values to generate “horizontal component” Gro 
values that approximate the Gro values generated by the G processing 
block 316 of the color-path module of Fig. 3 when the horizontal 
interpolation has been applied. 
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Thus, the mask used by the horizontal interpolation and averaging filter 
1206 is a 7x7 mask as follows: 
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In other words, the “horizontal component” Gro values generated by the 
horizontal interpolation and averaging filter 1206 represent the Gro values 
generated by the adaptive interpolator 320, the R sub-sampling unit 322 
and the interpolation and averaging filter 326 of the G processing block 
10 316 of Fig. 3 when the outputs of the adaptive interpolator 320 are the 

results of the horizontal interpolation. 

The vertical interpolation and averaging filter 1210 utilizes the 
following mask on the G2 values to generate “vertical component" Gro 
values that approximate the Gro values generated by the G processing 
15 block 316 of Fig. 3 when the vertical interpolation has been applied. 



”1/2~ 
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"averaging" 

mask 


1/2 







where the averaging mask is the same mask used by the horizontal 
20 interpolation and averaging filter 1206. Thus, the mask used by the 
vertical and averaging filter 1210 is a 7x7 mask as follows: 
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In other words, the “vertical componenf Gro values generated by the 
vertical interpolation and averaging filter 1210 represent the Gro values 
generated by the adaptive interpolator 320, the R sub-sampling unit 322 
5 and the interpolation and averaging filter 326 of the G processing block of 
Fig. 3 when the outputs of the adaptive interpolator 320 are the results of 
the vertical interpolation. 

The horizontal interpolation and averaging filter 1212 utilizes the 
same mask as the horizontal interpolation and averaging filter 1206 to 
io generate “horizontal component” Gro values that approximate the Gro 
values generated by the G processing block 316 of Fig. 3 when the 
horizontal interpolation has been applied. Similarly, the vertical 
interpolation and averaging filter 1210 utilizes the same mask as the 
vertical interpolation and averaging filter 1208 to generate "vertical 
15 componenf Gro values that approximate the Gro values generated by 
the G processing block 316 of Fig. 3 when the vertical interpolation has 
been applied. 

In operation, the G1-G2 separator 1204 receives the G intensity 
values within a given window of observation of an input mosaiced image. 
20 The G1-G2 separator transmits the G1 values of the G intensity values to 
the filters 1206 and 1208. In addition, the G1-G2 separator transmits the 
G2 values of the G intensity values to the filters 1210 and 1212. The 
horizontal interpolation and averaging filter 1206 generates Gro values 
that include horizontal interpolated components, while the vertical 
25 interpolation and averaging filter 1210 generates Gro values that include 
vertical interpolated components. These Gro values are received by the 
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selector 1214. The selector then transmits either the Gro values from the 
horizontal interpolation and averaging filter 1206 or the Gro values from 
the vertical interpolation and averaging filter 1210, depending on the 
determination of the pixel-wise gradient direction detector 506. 

5 Operating in parallel to the filters 1206 and 1210, the horizontal 

interpolation and averaging filter 1212 generates Gro values that include 
horizontal interpolated components, while the vertical interpolation and 
averaging filter 1208 generates Gro values that include vertical 
interpolated components. These Gro values are received by the selector 
10 1216. The selector then transmits either the Gro values from the 

horizontal interpolation and averaging filter 1212 or the Gro values from 
the vertical interpolation and averaging filter 1208, depending on the 
determination of the pixel-wise gradient direction detector 506. 

In Fig. 13, a G processing block 1302 having a more simplified 
15 configuration than the G processing block 1202 of Fig. 12 is shown. The 
7x7 masks used by the filters 1206-1212 of the G processing block 1202 
of Fig. 1 2 are similar to the original 5x5 averaging mask. That is, the 
central 5x5 portion of the 7x7 masks used by the filters 1206-1212 is 
similar to the original 5x5 averaging mask. Thus, the G processing block 
20 1302 of Fig. 13 approximates both of these 7x7 masks by the original 5x5 

averaging masks. As shown in Fig. 13, the G processing block 1302 
includes the G1-G2 separator 1204, the selectors 1214 and 1216, and the 
pixel-wise gradient direction detector 506, which are also found in the G 
processing module 1202 of Fig. 12. The only differences between the G 
25 processing modules of Figs. 12 and 13 are that the filters 1206 and 1208 
of the G processing module 1202 are replaced by an interpolation and 
averaging filter 1304 and the filters 1210 and 1212 of the G processing 
module 1202 are replaced by an interpolation and averaging filter 1306. 
The interpolation and averaging filter 1304 operates on the G1 values of a 
30 given window of observation of an input mosaiced image, while the 
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interpolation and averaging filter 1306 operates on the G2 values. These 
interpolation and averaging filters both use the original 5x5 averaging 
mask. The output values of the interpolation and averaging filter 1304 ^ . 

represent both the “horizontal component” Gro values and the “vertical 
5 componenf Gro values. Similarly, the output values of the interpolation 
and averaging filter 1 306 represent both the “horizontal componenf Gro 
values and the vertical componenf Gro values. Depending on the 
determination of the pixel-wise gradient direction detector 506, the 
selectors 1214 and 1216 transmit either the “horizontal componenf Gr 0 
io and Gro values or the “vertical componenf Gro and Gro values. * 

As stated previously, the demosaicing unit 104 provides color 
discontinuity equalization by enforcing the requirement that AR=AG=AB 
However , when strong color correction is applied after demosaicing, the 
resulting image is degraded. Under color correction of the type: 



~R'~ 
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= M ■ 


G 


B 




B_ 



20 



25 



where Mis a 3x3 matrix, the color discontinuity that will be observed are 
caused by linearity (A R‘, AG' or AB’), where 



~AR'~ 




'AR' 


AG' 


- M ■ 


AG 


A B' 




A B 



( 11 ) 



However , A R’, AG’ and AB’ will generally not satisfy the requirement that 
AR '=AG - AR’, which can result in artifacts. Thus, the color discontinuity 
equalization can be modified to satisfy the requirement of AR -AG - AR'. 

For color discontinuity equalization without color correction 
consideration, the color discontinuity is given by AG at a fixed pixel at a G 
location. The equalization is achieved by enforcing the equations, AR=AG 
and AB=AG, for demosaicing. These equations are modified to derive the 
following equations, which are more general. 

AR = a- AG ho\ 
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and 

AB = b-AG, (13) 

where a and b are fixed constants. For a given color correction matrix M, 
there exists a couple of unique constants (a, b) such that the above 
5 equations implies: 



A R'= AG' 


(14) 


AB'=AG'. 


(16) 



10 In addition, the couple of constants has a unique expression in terms of 
the coefficients of M. If the coefficients of Mare labeled as follows: 
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br 
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b b 



(16) 



15 then it can be shown by linear algebra that the unique solution in (a, b) has 
the following expressions. 

a = -{bg ■gb-bbgg-bgrb + gg-rb + bbrg-gb-rg)/D, (17) 

and 

b = -(br- gg — bg • gr -br-rb-V gr- rb + bb -rr — gg ■ rr)/ D, (18) 

20 where 

D = br • gb-bb- gr — br -rb + gr -rb + bb ■ rr — gb-rr . (19) 

Thus, the above expression can be used to provide color 
discontinuity equalization when demosaicing is followed by a strong color 
correction. In the case where the pixel is at an R location, the following 
25 equations are applied. 

AG = --AR (20) 

a 

and 
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AB = — ■ AR. 
a 

Similarly, at a B location, the following equations 

AR = -AB 
b 



( 21 ) 

are applied. 
( 22 ) 



and 






5 AG = ~ A£? . (23) 

This “color correction-compensated” equalization applies to a 
demosaicing process that uses equation (6) and (7), which are 
r = g+c ro and B = g+C b where C RQ = r 0 -g 0 and C B0 =B 0 -G 0 . 
However, for demosaicing process that uses equations (8) and (9), which 
10 are R = R 0 + A G R and B = B 0 + A G B , where AG R =G-G R0 and 

—G~ G bo , color discontinuity equalization needs to be further 
modified to satisfy the requirements of LR=AGr and A B = A Gr, where 
A Gr and A Gb are not necessarily equal. Since there are two values for 
the G color discontinuity, the transformation of color discontinuity as 
15 defined by expression (11) cannot be used. Thus, a different approach is 
used to satisfy the requirements of AR=AGr and AB = A Gr. 

For a given pixel location, A Gr and A Gb are assumed to have been 
calculated. The goal is to find A R and A B such that 

AR’-AG’r (24) 

20 and 

AB‘ = AG‘b (25) 

where 



A R' 




' AR~ 


AG' r 


— M • 


AG r 


donf care 




AB 



and 



(26) 
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'donf care" 




"A R' 


AG'b 


= M ■ 




AB' 




AB 



(27) 



In order to find a solution, the following equations, which are even more 
general than equations (12) and (13), are used. 

AR = a • AG R + b • AG B (28) 

and 



10 



15 



AB = c-AG R +d-AG B , (29) 

where the coefficients (a, b, c, d) are fixed constants. Again, it can be 



shown by linear algebra that by taking 



a b 
c d 



(bb - gb) • (gg - rg ) (bg -gg)(rb- g6)1 
(br-gr)-(rg-gg) (gg - bg) ■ (rr - gr) J 



the operations (28) and (29) satisfy equations (24) and (25). 

Thus, for each pixel location, the following equations are used. 
R = R 0 +a-AG R +b-AG B (30) 

B = B 0 +c-AG R +d-AG B (31) 



The above equations are derived from equations (8) and (9) using 
equations (28) and (29). 
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WHAT IS CLAIMED IS: 

1. A method of demosaicing a mosaiced image to derive a 
demosaiced image comprising: 

5 independently interpolating (304) first color values of said 

mosaiced image to derive first interpolated values of said demosaiced 
image; and 

interpolating (314 and 318) second color values of said 
mosaiced image to derive second interpolated values of said demosaiced 
10 image, including substantially equalizing (324, 334, 338 and 340) a 
discontinuity of said second interpolated values with a corresponding 
discontinuity of said first interpolated values. * 

2. The method of claim 1 wherein said step of independently 
15 interpolating (304) said first color values includes adaptively interpolating 
(310) said first color values using an interpolation technique selected from 
at least a first interpolation technique and a second interpolation 
technique. 

\ 

20 3. The method of claim 1 or 2 wherein said step of interpolating (314 

and 318) said second color values includes computing (316, 324, 334, 338 
and 340) color discontinuity equalization values using interpolated first 
color values and averaged first color values, said color discontinuity 
equalization values being used to substantially equalize said discontinuity 
25 of said second interpolated values with said corresponding discontinuity of ; 
said first interpolated values. 



< 
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4. The method of claim 3 wherein said step of computing (316, 324, 
334, 338 and 340) said color discontinuity equalization values includes: 

5 sub-sampling (706 and 712) said interpolated first color 

values with respect to pixel locations of said mosaiced image that 
correspond to said second color values of said mosaiced image to derive 
sub-sampled values; and 

averaging (708 and 714) said sub-sampled values to 
10 generate said averaged first color values. 

5. The method of claim 3 or 4 wherein said step of interpolating (314 
and 318) said second color values includes: 

averaging (330 and 336) said second color values of said 
15 mosaiced image to derived averaged - second color values; and 

summing (332, 334, 338 and 340) said color discontinuity 
equalization values and said averaged second color values to derive said 
second interpolated values. 

20 6. A system for demosaicing a mosaiced image to derive a 

demosaiced image comprising: 

a first interpolating module (304) for independently 
interpolating first color values of said mosaiced image to derive first 
interpolated values of said demosaiced image; 

25 a second interpolating module (314 and 318) for 

interpolating second color values of said mosaiced image to derive second 
interpolated values of said demosaiced image; and 

an equalization module (316, 332, 334, 338 and 340) for 
substantially equalizing a discontinuity of said second interpolated values 
30 with a corresponding discontinuity of said first interpolated values. 
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7. The system of claim 6 wherein said first interpolating module (304) 

includes an adaptive interpolator (310) that is configured to adaptively 
interpolate said first color values using an interpolation technique selected 
from at least a first interpolation technique and a second interpolation 
technique. ' 

8. The system of claim 6 and 7 wherein said equalization module 
(316, 332, 334, 338 and 340) is configured to compute color discontinuity 
equalization values (332, 334, 338 and 340) using interpolated first color 
values and averaged first color values, said color discontinuity 
equalization values being used by said equalization means to substantially . 
equalize said discontinuity of said second interpolated values with said 
corresponding discontinuity of said first interpolated values. 



9. The system of claim 8 wherein said equalization module (316, 332, 
334, 338 and 340) includes: 

a sub-sampling unit (322 and 324) that is configured to sub- * . 
sample said interpolated first color values with respect to pixel locations of 
said mosaiced image that correspond to said second color values of said 
mosaiced image to derive sub-sampled values; and 

an averaging unit (326 and 328) that is configured to 
average said sub-sampled values to generate said averaged first color 
values. 

10. The system of claim 8 or 9 wherein said second interpolating 
module (314 and 318) includes: 
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an averaging unit (330 and 336) that is configured to 
average said second color values of said mosaiced image to derived 
averaged second color values; and 

a computing unit (332, 334, 338 and 340) for summing said 
5 color discontinuity equalization values and said averaged second color 
values to derive said second interpolated values. 
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FIG. 1 
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FIG. 4 
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FIG. 5 
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FIG. 7 
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FIG. 8 
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FIG. 10 
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